System and method for online collaboration of synchronized audio and video data from multiple users through an online browser

ABSTRACT

A system and method for online collaboration of synchronized audio and video data from multiple users through an online browser is disclosed. An online network environment (e.g., online social media platform) that is accessible by users through an online browser includes an online player/recorder that is capable of playing back multiple audio and video data uploaded by multiple users in real time, and to change tempo while maintaining the original audio key or change pitch while maintaining the original tempo during playback. The platform captures playback positions of audio data based on the corresponding playback positions of video data. The platform includes a synchronizing algorithm that calculates the discrepancies between audio playback positions and the corresponding video playback positions so that the audio and video data play in synchronization.

RELATED APPLICATION

This application claims the benefit of priority under 35 U.S.C. §119(e)to Provisional Application No. 62/255,740, filed on Nov. 16, 2015, whichis incorporated by reference in its entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

INCORPORATION BY REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not applicable.

THE NAMES OP THE PARTIES TO A JOINT RESEARCH AGREEMENT

Not applicable.

BACKGROUND OF THE INVENTION

1. Field

The invention relates to a system and method for simultaneously playingback synchronized audio and video data from multiple users in an onlineenvironment, such as social media using Internet browsers. The inventionenables online users, through an online environment such as social mediathrough Internet browsers, to playback synchronized multiple audio andvideo data in real time, and to change the tempo during audio and videoplayback while maintaining the original audio key (i.e., pitch). Theinvention also enables online users to change the original audio key(i.e., pitch) during audio and video playback while maintaining perfectsynchronization between audio and video data.

2. Description of Related Art

In today's Internet age, it is often desirable to collaborate oncreating or playing back audio and video contents in an online virtualenvironment, for example social media using Internet browsers, becausecollaborating artists are often geographically separated and gatheringall collaborating artists at one location for creating, experiencing,practicing, and/or collaborating can be difficult and expensive. Forexample, in the context of music creation and performance, it is mucheasier and economically advantageous if musicians can collaborate as avirtual band.

Moreover, musicians often have varying levels of skill in differentaspects of music, such as skill level in different musical instruments,vocal abilities, music creativity, and varying perspective of music,such as music theory, composition, and other traits that affect theartistic rendition of music. Contributions from different musiciansoften lead to different music creations. Not every musician, however,knows or has access to all the music elements that may be contributed byother musicians. Thus, systems and/or methods that enable a user to posthis or her initial music creation on an online platform so that otherusers may access the initial music creation and contribute additionalmusic elements is desired.

Prior art discloses ideas of an online collaboration system whichenables users to post an idea on the online platform of a websitebrowser and enable other users to see the idea and give their commentsor feedbacks to the idea and rate the idea in a discussion section. Itis well known in the art to provide music uploading and downloading.Artists, including musicians, often look for sharing creative projectsthat utilizes audio and video expressions within a collaboration onlineenvironment so that they can reveal their projects, performances andattract other people to contribute and enjoy their projects (such as amusic creation). For example, U.S. Pat. No. 8,487,173 discloses amethods and systems for providing a collaborative music website whereusers can create and edit their music with others. This inventionincludes remotely uploading music tracks that are created off-line byusers, creating a band, and joining other users' band via an onlinenetwork. Other inventions also outline methods and/or systems forproviding an online collaborative website where a musician either join avirtual band with others or create their own virtual band to attractothers and edit music tracks remotely (i.e., off-line). Theseinventions, however, provide for only music (audio) collaboration, whichincludes multiple music tracks. No video collaboration is disclosed.Further, these inventions do not disclose online editing (such aschanging pitch or tempo) or online audio and video playback. When audioand video data are played back in an online environment in real time,audio and video synchronization is very important and difficult toachieve. It is difficult to playback a plurality of audios and videoswith perfect synchronization because the data formats between audio andvideo are different. This is a common issue in online playback thataudio and video would go out of sync, which may happen right at thebeginning or occur slowly over time. Many factors may cause thisproblem, including data transfer rate. Video generally requires moredata and longer time to process, matching video frame rate and audiosample rate in real time can be challenging. Furthermore, it isdifficult to simultaneously playback a plurality of video frames insynchronization because each video frame's data size being uploadedwithin one larger collaborative frame is individually different. Thesize of each video frame may be different depending on its individualdata amount used, even if the aspect ratio of each video frame isidentical. Data size of each individual video frame is based on theimage color and image movement within the uploaded video contents. Somevideo frames have more or less color saturation and speed of movementwhen playing. The present invention enables synchronization of themultiple individual frames with different Data size for each frame toplay as one combined frame. None of these prior inventions enables usersto upload and edit audio and video data with synchronization.

Therefore, there is a need for a system and method for onlinecollaboration of multiple tracks with audio and video datasynchronization. Because of this system and method, users may sharetheir creative works online while other users may view and listen to thecreative works online, contribute additional video and audio elements tothe creative works online, participating users may playback audio andvideo data in sync and in real time, and all users may edit music andaudio online (such as changing pitch and/or tempo) in real time.Separating and syncing multiple tracks of video and audio online enablesusers to bring their creative works and collaborate with people all overthe world.

BRIEF SUMMARY OF THE INVENTION

One of the uses for this invention is creating and editingcollaboratively multiple tracks of video and audio via an online networkplatform, for example, through Internet browsers. This platform enablesan online player/recorder to simultaneously playback a synchronizedplurality of video and audio tracks seamlessly. This includes capturingthe playback position of sound wave position (“MVP”) based on theplayback position of the video frame position (“VFP”). A synchronizingalgorithm Code was created that can calculate the discrepancy betweenthe SWP with the VFP and seamlessly adjusts the SWP and VFP when playedtogether simultaneously so that the audio and video will not go out ofsync.

To permit collaboration on a work-in-progress project (e.g., music, filmacting, dance, voice acting, etc.), a host user first uploads andposts/publishes the project on a website as a “open collaboration”project so that other users may participate in the same project.

For instance, when creating a song project with a single musicalinstrument (e.g., guitar, piano, drums, vocals), a host user uploads andposts an open collaboration project on a website and allowing otherusers to participate and collaborate within the project, for example, byrecording one or more parts using a different musical instrument (e.g.,vocal, bass, keyboard, drums). This can be accomplished by utilizing anonline player/recorder available within the website (accessible viaInternet browsers). Multiple musical instruments may be recorded onseparate tracks by other users. The uploaded and finished project can beedited, such as changing the tempo (speed) and key (pitch) of individualtracks, in real time, and perfect synchronization is maintained amongmultiple uploaded audios and videos while they are being viewed andplayed by online users.

These and other aspects of this invention will become apparent to thoseskilled in the art after reviewing the following description of theinvention.

BRIEF DESCRIPTION Of THE DRAWINGS

The above-mentioned features and objects of the present disclosure willbecome more apparent with reference to the following description takenin conjunction with the accompanying drawings wherein like referencenumerals denote like elements and in which:

FIG. 1 is a user interface that may be used to simultaneously playbackand edit a plurality of audio sound waves and video frames.

FIG. 2 shows a user interface that includes eight user windowsdisplaying audio and video. Each individual Frame data size isdifferent.

FIG. 3 is a graphic representation of a gap between the playbackposition of audio sound waves and video frames in normal tempo (speed).

FIG. 4 is a graphic representation of a mechanism and a formula toadjust the playback position of audio sound waves to the playbackposition of video frames in normal tempo (speed).

FIG. 5 is a graphic representation of a method to change video playbackto slow down the tempo (speed). Without changing the sound in real timefast or slow.

FIG. 6 is a graphic representation of a gap between the playbackposition of audio (sound waves) and video (frames) in a slower tempo(speed).

FIG. 7 is a graphic representation of a mechanism and a formula toadjust the playback position of audio (sound waves) to the playbackposition of video (frames) in a slower tempo (speed).

FIG. 8 is a graphic representation of an algorithm formula for audio(sound waves) and video (frames) playback and editing formula to adjustthe playback position when a cursor on the scroll bar is played back,paused, or moved.

FIG. 9 is a graphic representation of the mechanism to maintain the samekey (pitch) in a slower tempo (speed).

FIG. 10 is a graphic representation of a mechanism and a formula toadjust the playback position of audio sound waves to the playbackposition of video frames in normal tempo (speed) based on sound wavecalculation for CD data format quality, and an algorithm formula tosynchronize when a cursor on a scroll bar is moved, or paused.

FIG. 11 is a graphic representation and an algorithm formula forsynchronizing sound wave position and video frame position when thesound wave position creates a gap outside of the acceptable range oferror with respect to the video frame position.

FIG. 12 is a graphic representation of a user interface that includeseight user windows displaying audio and video, including data numbers,from each user in synchronization.

DETAILED DESCRIPTION OF THE INVENTION

For illustrative purpose, the principles of the present invention aredescribed by referring to an exemplary embodiment thereof Before anyembodiment of the invention is explained in detail, it should beunderstood that the invention is not limited in its application to thedetails of construction and the arrangement of components set forth inthe following description or illustrated in the following drawings. Theinvention is capable of other embodiments and of being practiced or ofbeing carried out in various ways. Also, it should be understood thatthe phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” or “having” and variations thereof herein ismeant to encompass the items listed thereafter and equivalents thereofas well as additional items.

The present invention focuses on online collaboration of audio and videobetween multiple users in social network platform. The method realizes aperfect synchronization of a plurality of audio and video data andenables users to simultaneously playback the plurality of audio andvideo data while maintaining the original key through an online browsernetwork. Users around the globe can create their own projects andcollaborate with other users who are interested in the existing projectsto share their works online.

FIG. 1 illustrates a graphic user interface of an Internet website,which may be accessible through Internet browsers, where a user canlogin to collaborate with other users and enable others to edit acollaborative project. Once the user log-in to the web page, the usercan watch a project, record their own project, join other users'projects, collaborate with other users to make a project, and/or edit aproject by changing the tempo (speed) and/or key (sound pitch) of afinished project. A plurality of audios and videos are shown in screen5, which enables users to arrange a ratio of small video screens. Usersmay manipulate actions (playback, stop, record, sound control) via icon6 and change the tempo (speed) and key (sound pitch) of a played videoon icon 7.

FIG. 2 illustrates a user interface that includes eight user windowsdisplaying audio and video for each user window. Each user window hasits own frame data. Each frame data size may be different. Data size ofeach frame is based on the image color and image movement of the contentwithin each user window. In one embodiment, synchronization of themultiple individual frames with different data size for each frame toplay as one combined master frame is calculated using the algorithm:Sound Position=8,192×(Video Position×5,3833)+6 (“SyncWave Algorithm”).In this embodiment, the SyncWave Algorithm calculates each frame tosynchronize as a whole when simultaneous playback is initiated. By wayof example, when individual musicians perform and play the same song andthe song is uploaded individually in one of eight frames and playback isviewed, the interface would show the song's audio and video playback isin sync with the other frames inside the master frame.

FIGS. 3 to 6 show a method for eliminating the gap deviation thatnaturally occurs when audio data (sound wave) is played with video data(frame) simultaneously. FIG. 3 is a graphic representation of a gapdeviation occurred between audio and video data. A recorded analog audiodata (sound wave) is going to be systematically converted to digitaldata and played in an online browser 10. The video data (frame) and itsnormal occurrence of the known disparity will be remedied as shown in 8.Based on how video data (frame) is normally converted against the audiodata (sound waves) in an online browser, the naturally occurring gapdeviation results in the audio sound data being delayed against thevideo data (frame) 9. Referring to FIG. 4, a synchronizing algorithmformula was created that can calculate and adjust the discrepancybetween the audio position (sound wave) with the video position (frame)and seamlessly adjusts 12 both the audio data (sound wave) with thevideo data (frame) 11 by making adjustment within the audio data (soundwave) 14 so that it plays along the video data (frame) 11. The formula13 shown in FIG. 4 shows an effective synchronization of the audio data(sound wave) and video data (frame) in normal tempo (speed). Thedisparity does not show when a user changes the playback tempo (speed),“var stint=(event.position)/8192×0.0027” is an embodiment of soundsyncing algorithm (algorithm is coded using ActionScript).

Referring now to FIG. 5, a method for changing a playback tempo (speed)of a pluralized audio data (sound wave) and video data (frame) that aresimultaneously played is described. Inserting a pause in the gapdeviation when adjusting the tempo (speed) to a slower speed achievesthe synchronization of changing audio data (sound wave) and video data(frame), i.e., eliminating the gap deviation. Sound is heard in realtime against the slow video image. (normal frame per second in video iscalculated at about 30 frames per second). Slowing down the video, theSyncWave Algorithm fills in the gaps by creating pauses in place ofempty frames. By way of example, 2 times slower creates double the frameratio to show about 30 frames plus 30 frames of pauses per 2 seconds inslowing down the video image; and 3 times slower create triple the frameratio to show about 30 frames plus 60 frames of pauses per 3 seconds inslowing down the video image.

FIG. 6 illustrates what would happen to the audio data (sound wave) andvideo data (frame) when playback tempo (speed) is changed to slow. Inone embodiment, the audio data (sound wave) 17 continuously plays asnormal. However the video data (frame) 15 is not playing normally due tothe change in speed. A gap occurs between the playback position of audio(sound waves) 17 and video (frames) 15 in a slower tempo (speed).Because of the synchronizing algorithm formula's automatic adjustment,videos are without video frame delays, stutters and/or distortion duringplayback. Distortion 16 is caused when releasing the memory, which isaccumulated through the high-speed processing in making the playbackslow.

FIG. 7 describes a method and formula for adjusting gap deviation whenan irregular video data (frame) interacts with normal audio data (soundwave) as well as when a normal video data (frame) 18 plays with anirregular audio data (sound wave) 19. As shown in FIG. 7, a mechanismand a formula adjust an irregular audio data (sound) 20 to the same asthe video data (frame) by adjusting the playback position of audio(sound waves) 19 to the playback position of video (frames) 18 in aslower tempo (speed). A synchronizing algorithm formula 21 is used tocalculate how much sound data needs to he fixed in order to adjust tovideo data (frame). “i+1” is a program coding algorithm name foradjusting audio speed to make it go faster; and “(j+1)x−1” is a programcoding algorithm name for adjusting audio speed to make it go slower.

FIG. 8 illustrates a method and formula for pausing and adjusting aplayback position for audio sound and video frames when playback ispaused or a cursor on a scroll bar is moved (e.g., the seek function).Whenever a user uses the cursor to watch a particular part of what theyare looking for and moves the playback position of the audio and video,the audio data stops immediately. However the video data slips forwardoff its track from its regular position. The SyncWave Algorithmmaintains a synchronized playback position for the audio and videoimmediately.

The key (pitch) of a sound is changed automatically when thecorresponding tempo (speed) is changed to slow without adjustment. FIG.9 is a graphic representation that illustrates how the key (pitch) ofaudio data (sound) goes to a slow tempo (speed) and how it isautomatically adjusted to become a regular wavelength. When a wavelengthof an audio data (sound) 27 is played at a normal tempo 26, it is playednormally and maintains the original key. However, once the tempo (speed)is slowed down 25, the wavelength of the audio data (sound) bends andgoes off from the original key (pitch) as described in 28. Thesynchronizing algorithm formula, automatically adjusts the bentwavelength of the audio data (sound) to the original key (pitch) 29.

Similar to FIG. 4, FIG. 10 shows an effective synchronization of theaudio data (sound wave) and video data (frame). FIG. 10 describes theSyncWave Algorithm (Sound Position=8,192×(Video Position×5.3833)+6) forsynchronizing audio and video data when a cursor is moved or playback isstopped or temporarily paused. In one embodiment, a mechanism and aformula is provided to adjust the playback position of audio sound wavesto the playback position of video frames in normal tempo (speed) basedon sound wave calculation for CD data format quality. An algorithmformula (var pp=(nStream[0].time×53833)+6) for seeking the correctposition to match audio and video is also provided to synchronize when acursor on a scroll bar is moved or video playback is paused.

FIG. 11 illustrates an algorithm formula (coded in ActionScript) forsynchronize SWP and VFP when the SWP creates a gap outside of theacceptable range of error with respect to the VFP.

FIG. 12 illustrates a user interface that includes eight user windowsdisplaying both audio and video, including data numbers, from each userin synchronization. For each user, the audio and video synchronizationfor each user is displayed, 100 indicates perfect synchronization.

The previous description of the disclosed examples is provided to enableany person of ordinary skill in the art to make or use the disclosedmethods and apparatus. Various modifications to these examples will bereadily apparent to those skilled in the art, and the principles definedherein may be applied to other examples without departing from thespirit or scope of the disclosed method and apparatus. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive and the scope of the invention is, therefore,indicated by the appended claims rather than by the foregoingdescription. All changes, which come within the meaning and range ofequivalency of the claims, are to be embraced within their scope.Skilled artisans may implement the described functionality in varyingways for each particular application, but such implementation decisionsshould not be interpreted as causing a departure from the scope of thedisclosed apparatus and methods. The steps of the method or algorithmmay also be performed in an alternate order from those provided in theexamples.

1. A method for synchronizing audio and video data from multiple usersin an online network environment comprising: uploading by a host user aproject to an online network environment that permits access to theproject by one or more other users, wherein the online networkenvironment comprising an audio and video player/recorder that enablesthe host user and the one or more other users to upload, download,record, and playback audio and video data within the online networkenvironment in real time; the project comprising one or more audio dataand one or more video data wherein the one or more audio data comprisingone or more audio playback positions, and the one or more video datacomprising one or more video playback positions; each of the one or moreaudio playback positions temporally corresponds to at least one of theone or more video playback positions; providing a synchronizingalgorithm configured to capture the one or more audio playback positionsand the one or more video playback positions, and to temporally adjustthe one or more audio playback positions and/or the one or more videoplayback positions so that the one or more audio playback positions andthe one or more video playback positions are substantially in synctemporally.
 2. A method for synchronizing audio and video data frommultiple users in an online network environment according to claim 1wherein the synchronizing algorithm is configured to capture and adjustthe one or more audio playback positions and the one or more videoplayback positions in real time.
 3. A method for synchronizing audio andvideo data from multiple users in an online network environmentaccording to claim 1 further comprising the step of providing analgorithm formula configured to adjust the one or more audio data andthe one or more video data when playback is paused, sped up, or sloweddown.
 4. A system for synchronizing audio and video data from multipleusers in real time within an online network environment comprising: auser interface configured to allow a host user uploading a project to anonline network environment that permits access to the project by one ormore users; the user interface further comprising an audio and videoplayer/recorder configured to enable the host user and the one or moreusers to upload, download, record, and playback audio and video data inthe online network environment in real time; the project comprising oneor more audio data and one or more video data wherein the one or moreaudio data comprising one or more audio playback positions and one ormore video data comprising one or more video playback positions; each ofthe one or more audio playback positions temporally corresponds to atleast one of the one or more video playback positions; a synchronizingalgorithm configured to capture the one or more audio playback positionsand the one or more video playback positions, and to temporally adjustthe one or more audio playback positions and/or the one or more videoplayback positions so that the one or more audio playback positions andthe one or more video playback positions are substantially in synctemporally.
 5. A system according to claim 4 wherein the user interfacefurther comprising one or more user windows wherein each of the one ormore user windows is configured to playback audio and/or video data fromthe host user or one of the one or more other users.
 6. A systemaccording to claim 4 wherein the synchronizing algorithm is configuredto capture and adjust the one or more audio playback positions and theone or more video playback positions in real time.
 7. A system accordingto claim 4 further comprising an algorithm formula configured to adjustthe one or more audio data and the one or more video data when playbackis paused, sped up, or slowed down